Notes on Pca, Regularization, Sparsity and Support Vector Machines

We derive a new representation for a function as a linear combination of local correlation kernels at optimal sparse locations and discuss its relation to PCA, regularization, sparsity principles and Support Vector Machines. We also discuss its Bayesian interpretation and justiication. We rst review previous results for the approximation of a function from discrete data (Girosi, 1998) in the context of Vapnik's feature space and dual representation (Vapnik, 1995). We apply them to show 1) that a standard regularization functional with a stabilizer deened in terms of the correlation function induces a regression function in the span of the feature space of classical Principal Components and 2) that there exist a dual representations of the regression function in terms of a regularization network with a kernel equal to a generalized correlation function. We then describe the main observation of the paper: the dual representation in terms of the correlation function can be sparsiied using the Support Vector Machines (Vapnik, 1982) technique and this operation is equivalent to sparsify a large dictionary of basis functions adapted to the task, using a variation of Basis Pursuit In all cases { regularization, SVM and BPD { we show that a bayesian approach jus-tiies the choice of the the correlation function as kernel. In addition to extending the close relations between regularization, Support Vector Machines and sparsity, our work also illuminates and formalizes the LFA concept of Penev and Atick (1996). We discuss the relation between our results, which are about regression, and the diierent problem of pattern classiication.